We are using a two-component hurdle model: first, the model predicts whether a disease will be present (binary), and if present, it predicts the case count (integer). Here we compare the results of a boosted tree model to our baseline model.
| .metric | desc | model | full_model |
|---|---|---|---|
| accuracy | proportion of the data that are predicted correctly | baseline | 0.85 |
| xgboost | 0.96 | ||
| kap | similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. | baseline | 0.45 |
| xgboost | 0.88 | ||
| sens | the proportion of positive results out of the number of samples which were actually positive. | baseline | 0.99 |
| xgboost | 0.98 | ||
| spec | the proportion of negative results out of the number of samples which were actually negative | baseline | 0.36 |
| xgboost | 0.88 |
| .metric | model | birds | buffaloes | camelidae | cats | cattle | cervidae | dogs | equidae | hares/rabbits | sheep/goats | swine |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| accuracy | baseline | 0.85 | 0.77 | 0.780 | 0.76 | 0.86 | 0.730 | 0.78 | 0.90 | 0.85 | 0.86 | 0.87 |
| xgboost | 0.95 | 0.96 | 0.960 | 0.96 | 0.95 | 0.960 | 0.95 | 0.97 | 0.96 | 0.96 | 0.96 | |
| kap | baseline | 0.42 | 0.21 | 0.130 | 0.36 | 0.57 | 0.084 | 0.49 | 0.42 | 0.19 | 0.46 | 0.43 |
| xgboost | 0.85 | 0.91 | 0.890 | 0.92 | 0.88 | 0.910 | 0.89 | 0.87 | 0.86 | 0.88 | 0.87 | |
| sens | baseline | 0.98 | 1.00 | 1.000 | 1.00 | 0.99 | 1.000 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
| xgboost | 0.98 | 0.98 | 0.980 | 0.98 | 0.98 | 0.980 | 0.97 | 0.99 | 0.98 | 0.98 | 0.98 | |
| spec | baseline | 0.34 | 0.15 | 0.094 | 0.30 | 0.50 | 0.061 | 0.45 | 0.31 | 0.14 | 0.37 | 0.33 |
| xgboost | 0.86 | 0.92 | 0.910 | 0.94 | 0.89 | 0.920 | 0.91 | 0.85 | 0.85 | 0.89 | 0.86 |
| .metric | model | Africa | Americas | Asia | Europe | NA | Oceania |
|---|---|---|---|---|---|---|---|
| accuracy | baseline | 0.83 | 0.82 | 0.85 | 0.87 | 0.94 | 0.930 |
| xgboost | 0.95 | 0.96 | 0.96 | 0.95 | NA | 0.990 | |
| kap | baseline | 0.47 | 0.39 | 0.47 | 0.47 | 0.51 | 0.094 |
| xgboost | 0.88 | 0.91 | 0.89 | 0.83 | NA | 0.890 | |
| sens | baseline | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 1.000 |
| xgboost | 0.98 | 0.98 | 0.98 | 0.98 | NA | 1.000 | |
| spec | baseline | 0.39 | 0.31 | 0.38 | 0.38 | 0.42 | 0.054 |
| xgboost | 0.89 | 0.92 | 0.89 | 0.83 | NA | 0.870 |
| .metric | desc | model | full_model |
|---|---|---|---|
| accuracy | proportion of the data that are predicted correctly | baseline | 0.850 |
| xgboost | 0.960 | ||
| kap | similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. | baseline | 0.046 |
| xgboost | 0.500 | ||
| sens | the proportion of positive results out of the number of samples which were actually positive. | baseline | 0.460 |
| xgboost | 0.560 | ||
| spec | the proportion of negative results out of the number of samples which were actually negative | baseline | 0.680 |
| xgboost | 0.790 |
| .metric | model | birds | buffaloes | camelidae | cats | cattle | cervidae | dogs | equidae | hares/rabbits | sheep/goats | swine |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| accuracy | baseline | 0.850 | 0.770 | 0.780 | 0.760 | 0.860 | 0.73 | 0.7800 | 0.900 | 0.850 | 0.860 | 0.870 |
| xgboost | 0.950 | 0.960 | 0.960 | 0.960 | 0.950 | 0.96 | 0.9500 | 0.970 | 0.960 | 0.960 | 0.960 | |
| kap | baseline | 0.071 | 0.017 | 0.048 | -0.028 | 0.042 | -0.02 | -0.0014 | 0.063 | 0.026 | 0.044 | 0.065 |
| xgboost | 0.420 | 0.640 | 0.640 | 0.720 | 0.460 | 0.72 | 0.5900 | 0.470 | 0.500 | 0.510 | 0.420 | |
| sens | baseline | 0.450 | 0.560 | 0.580 | 0.610 | 0.430 | 0.59 | 0.4700 | 0.480 | 0.420 | 0.460 | 0.500 |
| xgboost | 0.520 | 0.610 | 0.600 | 0.560 | 0.540 | 0.59 | 0.5700 | 0.570 | 0.560 | 0.550 | 0.520 | |
| spec | baseline | 0.690 | 0.650 | 0.670 | 0.630 | 0.670 | 0.62 | 0.6500 | 0.690 | 0.670 | 0.680 | 0.690 |
| xgboost | 0.760 | 0.840 | 0.840 | 0.880 | 0.780 | 0.88 | 0.8300 | 0.790 | 0.790 | 0.800 | 0.770 |
| .metric | model | Africa | Americas | Asia | Europe | NA | Oceania |
|---|---|---|---|---|---|---|---|
| accuracy | baseline | 0.830 | 0.820 | 0.850 | 0.870 | 0.940 | 0.930 |
| xgboost | 0.950 | 0.960 | 0.960 | 0.950 | NA | 0.990 | |
| kap | baseline | 0.024 | 0.014 | 0.054 | 0.088 | -0.019 | 0.065 |
| xgboost | 0.520 | 0.550 | 0.540 | 0.400 | NA | 0.460 | |
| sens | baseline | 0.450 | 0.450 | 0.470 | 0.470 | 0.330 | 0.590 |
| xgboost | 0.550 | 0.560 | 0.590 | 0.520 | NA | 0.540 | |
| spec | baseline | 0.660 | 0.660 | 0.680 | 0.690 | 0.660 | 0.710 |
| xgboost | 0.800 | 0.810 | 0.810 | 0.760 | NA | 0.780 |
Here we evaluate the subset of the training data with positive case counts
cases model stats
## # A tibble: 6 x 4
## model .metric .estimator .estimate
## <chr> <chr> <chr> <dbl>
## 1 baseline rmse standard 105536.
## 2 xgboost rmse standard 285638.
## 3 baseline rsq standard 0.907
## 4 xgboost rsq standard 0.540
## 5 baseline mae standard 1234.
## 6 xgboost mae standard 2231.